The inputting of information into electronic systems such as personal computers (PCs), mobile phones, palm computers, aircraft computers, etc., using input devices with keys, for example keyboards, touch pads or touch screens, all henceforth referred to as xe2x80x9ckeyboardsxe2x80x9d, has not changed significantly since the invention of the PC. Most of the information or data is typed using a keyboard with fixed key functions. Particularly with PCs, dual language keyboards normally have a second language alphabet symbols etched into the keyboard keys together with the English alphabet, with the switching between the dual language functions of each key achieved generally by software instructions. In a standard keyboard, a user usually views up to three symbols imposed on each key, defining different languages and options provided by different software and system operation methods. The mouse also serves independently for graphic command inputs.
Standard keyboards suffer from a number of disadvantages and limitations. They normally contain function keys, for example with symbols F1 to F12, which hide functions defined in separate instructions. The user has to learn and memorize these hidden functions, or constantly invoke their meaning from a look-up table, a xe2x80x9cHelpxe2x80x9d directory or other sources. Such keyboards are limited in the number of keys, and therefore key functions. A non-professional typist normally has to follow the typing action by shifting frequently his gaze between the keyboard placed on a desk and the PC monitor screen, which is normally placed in front and higher on the desk. In particular with dual-language keyboards, this frequent eye shifting, and the frequent non-feedback use of the xe2x80x9cAlt+Shiftxe2x80x9d and xe2x80x9cCaps Lockxe2x80x9d functions, leads to errors in typing. With the advent of wide Internet usage, PC users are required to use the standard keyboard for more and more complicated input commands, or for remembering more xe2x80x9chiddenxe2x80x9d functions of each key.
Advanced keyboards with changeable key functions, and in particular keyboards using keyboard emulation, virtual keyboards, and other techniques to overcome some of the limitations mentioned above, have been proposed in the past. For example, U.S. Pat. Nos. 4,431,988, 4,551,717, 4,633,227, 4,647,911, and 5,164,723 propose keyboards with changeable key displays or definable keys, mostly implemented in the hardware keyboard itself. However, none of these patents or other prior art changeable or configurable keyboards provide a keyboard which has fully selectable, configurable and changeable keys imposed on the virtual representation of a keyboard, with the changeable key functions provided by various sources, as does the present invention. U.S. Pat. No. 5,164,723 for example, reviews prior art technologies regarding hardware keyboards with changeable key displays, and describes a changeable keys keyboard which has a plurality of key operated switches, each including a key having a manually engageable key surface. The display on the key is changed by means such as optical fibers or LCDs, in some cases automatically according to the present function of a key. This is a xe2x80x9chardwarexe2x80x9d implementation of a changeable keyboard, which makes the keyboard cumbersome and expensive. The keys can be changed by user instructions only, not by software instructions from other sources such as applications software and Internet sites, as provided by a main feature of the present invention.
A recent U.S. Pat. No. 6,037,882, discloses a method and apparatus for inputting data to an electronic system using virtual reality techniques. Virtual reality is used primarily in some electronic and computer games. Virtual reality simulation is usually achieved by the use of wearable computing technology, typically sophisticated computers having communication ability to the bodies of the users. The output from a wearable computer is normally viewed on a display, which may be worn over the eye of a user. The main objective of U.S. Pat. No. 6,037,882 is to allow switching between two modes of an input device such as a keyboard, without having the user look at the real device, and instead having him look at a virtual representation of the device. The method of U.S. Pat. No. 6,037,882 includes the steps for generating a representation of an input device, having a plurality of input options in virtual space, sensing the position of a user in real-space, using a surface sensor, and displaying the representation of the input device and the position of the user in virtual space on a head-mounted display. However, in contrast with the present invention, this patent employs a fixed configuration, standard keyboard that cannot be assigned changeable keys, i.e. other functions, and therefore lacks the versatility that is one of the central features of the present invention. Sensors are incorporated in the keyboard itself, making it expensive. There is a very basic simulation of the hands, but no full simulation of both hands and keyboard, and the input (keying) action is not a natural action of typing, as in the present invention. The preferred input action in U.S. Pat. No. 6,037,882 is an xe2x80x9cunnaturalxe2x80x9d (to a typist) action, in which a cursor-like finger presses one key, when the input action itself is governed by a dedicated microprocessor. Therefore, U.S. Pat. No. 6,037,882 lacks a few of the basic advantages of the present invention, for example the advantage that the present invention enables a natural typing action as commonly practiced, using all ten life-like fingers of two hands simulated in real-time together with the keyboard on a display.
U.S. Pat. No. 6,088,018 describes the use of a video camera or sensor to image all or parts of a user""s body on a display, in order to have an interactive data input action in the display. The user image is a video image, not a virtual one, and there is no specific mention of an external input device. The essence of the method is a new way of inputting data xe2x80x9cin the displayxe2x80x9d, not of simulating a virtual reality image of a well established input device such as a keyboard, which then can be viewed, assigned any desired function to any of its keys, and activated in a natural way by natural typing action when the user sees his hands on the same display. The method of U.S. Pat. No. 6,088,018 does not view a keyboard as an important element, does not provide therefore the changeable keys and function assignation features as does the present invention, and requires the development of new, expensive, and complicated devices.
There is thus a widely recognized need for, and it would be highly advantageous to have, a method and system for inputting data into an electronic system using a keyboard with changeable keys, which a user can activate by natural typing movements. Specifically, there is a need for, and would be advantageous to have a method and system to simulate in real-time and view a virtual keyboard on a screen or display, to assign any key or function to the virtual keyboard, to simulate in real-time and view the hands of a user positioned over the virtual keyboard, and to use the virtual keyboard as if it is a real keyboard. The method and system of the present invention allow a real hardware keyboard which is scanned in order to obtain the simulation to be blank, simple, cheap and universal, with the assignation of key functions to virtual keys of the virtual keyboard implemented via software instructions and available from various sources, particularly applications software and Internet sites.
According to the present invention, there is provided a method for inputting data into an electronic system, comprising the steps of: providing a keyboard connected to the system, forming a virtual image of the keyboard on a display of the electronic system, the virtual image functioning as a virtual keyboard, assigning at least one symbol to at least one key of the virtual keyboard, forming a virtual image of at least one hand in spatial relationship to the virtual keyboard, the virtual image functioning coordinately with a hand of a user of the system such that it functions as a virtual hand of the user, and inputting data via the keyboard with the at least one hand.
According to one feature in the method of the present invention, the steps of forming a virtual image at least one hand and of the keyboard comprise the sub-steps of scanning the at least one hand and the keyboard, transmitting the information obtained from the scanning to the electronic system, and simulating in real-time the at least one virtual hand and the virtual keyboard.
According to another feature in the method of the present invention, the step of inputting further includes the sub-step of positioning the at least one virtual hand over the virtual keyboard, and influencing the at least one key to generate an input signal into the system.
According to yet another feature in the method of the present invention, the step of simulating further comprises running a calibration procedure for the at least one virtual hand and the virtual keyboard.
According to yet another feature in the method of the present invention, the step of scanning is performed by at least one scanner.
According to yet another feature in the method of the present invention, the step of assigning further comprises: providing a changeable keys software to the system, and instructing the changeable keys software to assign the at least one symbol.
According to yet another feature in the method of the present invention, the scanning by the at least one scanner includes sensing at least one member of the group consisting of contrast, temperature, color, shape, slant, sound, electromagnetic radiation, volume changes, humidity, static air pressure, size, motion, and Doppler effect.
According to yet another feature in the method of the present invention, the step of instructing is performed manually.
According to yet another feature in the method of the present invention, the step of the providing includes supplying an exchange file, and the step of instructing includes receiving inputs from external sources.
According to yet another feature in the method of the present invention, the step of instructing further includes choosing the at least one symbol selected from the group consisting of color, size, language, function, letters, words, sentences, animation, digital display, video images and voice.
According to yet another feature in the method of the present invention, the external sources include applications software and Internet sites.
According to the present invention, there is provided in an electronic system, a virtual input environment system comprising: at least one scanner for scanning real-space objects, the real-space objects including at least one hand and an input device, the at least one scanner connected to the system, a simulation software for processing information obtained from the at least one scanner, and for generating virtual images of the at least one hand and the input device, a changeable keys software for assigning virtual keys to the virtual image of the input device, a display for displaying the virtual images of the at least one hand and the input device, wherein the at least one hand can be used to input data through the input device into the electronic system.
According to one feature in the virtual input environment of the present invention, the input device includes a keyboard.
According to another feature in the virtual input environment of the present invention, the display is selected from the group consisting of computer monitor, eye-mounted display, television screen, head-up display and eye oriented display.
According to yet another feature in the virtual input environment of the present invention, the virtual image of the keyboard is configurable in a plurality of configurations.
According to yet another feature in the virtual input environment of the present invention, the keyboard includes a blank keyboard.
According to yet another feature in the virtual input environment of the present invention, the keyboard includes a flat substrate.
According to yet another feature in the virtual input environment of the present invention, the keyboard includes a sensing plate.
According to yet another feature in the virtual input environment of the present invention, the sensing plate includes sensing elements chosen from one of the group consisting of pressure, heat, and sound.
According to yet another feature in the virtual input environment of the present invention, the keyboard comprises two parts.
According to yet another feature in the virtual input environment of the present invention, the plurality of configurations includes changeable characters selected from the group consisting of color, size, language, function, letters, words, sentences, animation, digital display, video images and voice.